Skip to content

feat(iota-core): introduce StoreObjectV2 and snapshot V2#11453

Open
bingyanglin wants to merge 2 commits intodevelopfrom
feat/iota-core-storeobjectv2-and-epoch-info
Open

feat(iota-core): introduce StoreObjectV2 and snapshot V2#11453
bingyanglin wants to merge 2 commits intodevelopfrom
feat/iota-core-storeobjectv2-and-epoch-info

Conversation

@bingyanglin
Copy link
Copy Markdown
Contributor

@bingyanglin bingyanglin commented May 7, 2026

Description of change

Make new generated formal snapshots carry per-epoch metadata and per-object checkpoint attribution, without requiring archival replay.

Storage schema (iota-core, iota-types)

  • New StoreObjectV2 adds previous_transaction_checkpoint: CheckpointSequenceNumber to each stored object value. V1 rows lift to V2 lazily on read via StoreObjectWrapper::migrate(). SENTINEL_PREVIOUS_TRANSACTION_CHECKPOINT = u64::MAX marks rows whose containing checkpoint is not yet known (PassthroughCache write path; V1 → V2 lifts; restore path).
  • New epoch_info: DBMap<EpochId, EpochInfoEntry> column family on CheckpointStoreTables, sibling of epoch_last_checkpoint_map. Holds { last_checkpoint_summary: CertifiedCheckpointSummary, first_checkpoint, end_of_epoch_tx_events: TransactionEvents } per epoch — schema confirmed by indexer team.
  • EpochInfoEntry lives in a new iota-types::epoch_info module so producer (iota-core) and consumer (iota-snapshot) share the type.

Execution path (iota-core)

  • ExecutionCacheCommit::build_db_batch gains a checkpoint_seq: CheckpointSequenceNumber parameter, threaded from the executor → WritebackCache::build_db_batchAuthorityStore::build_db_batchwrite_one_transaction_outputsget_store_object.
  • Cache-layer asymmetry:
    • WritebackCache (default config) buffers tx outputs and flushes at commit_transaction_outputs; the executor passes the real seq there, so newly-written rows on default-deployed nodes carry real values.
    • PassthroughCache writes synchronously before the containing checkpoint is determined; stamps the sentinel until PR 2's backfill rewrites those rows.
  • New CheckpointExecutor::write_epoch_info_entry fires inside the executor's is_final_checkpoint block (alongside insert_epoch_last_checkpoint and accumulate_epoch). It runs after commit_transaction_outputs, so AdvanceEpoch events are durable when read.

Snapshot V2 file format (iota-snapshot)

  • Reference records grow from 72 to 80 bytes: V1 layout + 8-byte big-endian previous_transaction_checkpoint trailer. New magic REFERENCE_FILE_MAGIC_V2 = 0xCAFEBEEF (V1 was 0xDEADBEEF); a V1 reader fails fast on the magic.
  • New per-snapshot EPOCH_INFO file carrying EpochInfo::V1(EpochInfoV1 { entries: Vec<Option<EpochInfoEntry>> }), length snapshot_epoch + 1. None entries indicate the producer had no row for that epoch (typical only during the PR 1 → PR 2 window). Magic 0x9000C001. Sha3-anchored via FileMetadata::sha3_digest in the MANIFEST.
  • Manifest version bumped to V2. Manifest::V1 and Manifest::V2 share ManifestBody; the BCS variant tag distinguishes them.
  • V2-only reader: older V1 snapshots are rejected up-front with a clear error. Operators regenerate snapshots after PR 1 deploys (next epoch boundary on a V2-running node produces a V2 snapshot).
  • New LiveSetIterV2 on AuthorityPerpetualTables yields (LiveObject, CheckpointSequenceNumber) pairs so the writer can emit the per-record trailer without an extra DB lookup.

Cleanup (iota-core, iota-snapshot, iota-single-node-benchmark, iota-surfer)

  • LiveObject::Wrapped variant removed. Pre-merge proof: LiveSetIter::store_object_wrapper_to_live_object only returns LiveObject::Normal or None, and a workspace-wide grep confirmed zero construction sites — every LiveObject::Wrapped reference was a destructure on input the iterator could never produce. Eight match arms across four crates simplified or eliminated. Orphaned WrappedObject helper in global_state_hasher.rs removed.

Links to any relevant issues

fixes #11254

How the change has been tested

  • Basic tests (linting, compilation, formatting, unit/integration tests)
  • Patch-specific tests (correctness, functionality coverage)
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that new and existing unit tests pass locally with my changes

@bingyanglin bingyanglin self-assigned this May 7, 2026
@bingyanglin bingyanglin added the node Issues related to the Core Node team label May 7, 2026
@bingyanglin bingyanglin force-pushed the feat/iota-core-storeobjectv2-and-epoch-info branch from dd98261 to e31f6a0 Compare May 7, 2026 09:34
@bingyanglin bingyanglin marked this pull request as ready for review May 7, 2026 09:34
@bingyanglin bingyanglin requested review from a team as code owners May 7, 2026 09:34
@muXxer muXxer added this to the v1.24.0 milestone May 7, 2026
@bingyanglin bingyanglin force-pushed the feat/iota-core-storeobjectv2-and-epoch-info branch from 99043df to 23753dd Compare May 8, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core-protocol node Issues related to the Core Node team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

snapshots: extended formal snapshot format

3 participants